Incident Response Process & Postmortem Template
Overview
This document outlines the incident escalation process and blameless postmortem template for production incidents on Learnille.
🚨 1. Incident Severity Levels
| Severity | Definition | Response SLA | Target Resolution |
|---|---|---|---|
| SEV-1 (Critical) | Core service outage (e.g. payment gateway failing, database down, total API failure). | < 15 mins | < 2 hours |
| SEV-2 (Major) | Major feature degraded (e.g. video streaming buffering, notification delays). | < 1 hour | < 6 hours |
| SEV-3 (Minor) | Non-critical bug (e.g. cosmetic UI glitch, analytics logging drop). | Next business day | Next sprint release |
📝 2. Blameless Postmortem Template
# Incident Postmortem: [INC-YYYY-MM-DD-ShortName]
## Executive Summary
- **Date & Time**: YYYY-MM-DD HH:MM UTC
- **Duration**: XX minutes
- **Severity**: SEV-1 / SEV-2
- **Impact**: XX% of users affected. XX failed course purchases / booking attempts.
- **Root Cause**: Short description of failure mechanism.
## Timeline (All times in UTC)
- **10:00** - Automated Sentry alert triggered for high error rate on `/payments/webhook`.
- **10:05** - On-call engineer acknowledged incident and began log investigation.
- **10:20** - Mitigating hotfix deployed to staging and verified.
- **10:35** - Hotfix deployed to production; error rate returned to normal.
## Root Cause Analysis (5 Whys)
1. **Why did the payment webhook fail?** Signature verification failed due to missing environment variable.
2. **Why was the env variable missing?** Deployment script overwrote environment configuration during deployment.
## Corrective Actions & Preventative Tasks
- [ ] Add CI environment variable validation check before deployment script execution.
- [ ] Add automated E2E test for webhook signature verification.